Optimizing process allocation of parallel programs for heterogeneous clusters

The performance of a conventional parallel application is often degraded by load-imbalance on heterogeneous clusters. Though it is simple to invoke multiple processes on fast PEs to alleviate loadimbalance, the optimal process allocation is not obvious. Kishimoto and Ichikawa presented performance models for HPL (High Performance Linpack), with which the sub-optimal configurations of heterogeneous clusters were actually estimated. Their results on HPL are encouraging, whereas their approach is not yet verified with other applications. This study presents some enhancements of Kishimoto’s scheme, which are evaluated with four typical scientific applications: CFD (computational fluid dynamics), FEM (finite element method), HPL (linear algebraic system), and FFT (fast Fourier transform). According to our experiments, our new models (NP-T models) are superior to Kishimoto’s models, particularly when non-negative least squares (NNLS) method is used for parameter extraction. The average errors of the derived models were 0.2% for CFD benchmark, 2% for FEM benchmark, 1% for HPL, and 28% for FFT benchmark. This study also emphasizes the importance of predictability in clusters, listing practical examples derived from our work.